15 research outputs found
Momentum in Reinforcement Learning
We adapt the optimization's concept of momentum to reinforcement learning.
Seeing the state-action value functions as an analog to the gradients in
optimization, we interpret momentum as an average of consecutive -functions.
We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that
incorporates this momentum idea. Our analysis shows that this allows MoVI to
average errors over successive iterations. We show that the proposed approach
can be readily extended to deep learning. Specifically, we propose a simple
improvement on DQN based on MoVI, and experiment it on Atari games.Comment: AISTATS 202
Evaluating the Moral Beliefs Encoded in LLMs
This paper presents a case study on the design, administration,
post-processing, and evaluation of surveys on large language models (LLMs). It
comprises two components: (1) A statistical method for eliciting beliefs
encoded in LLMs. We introduce statistical measures and evaluation metrics that
quantify the probability of an LLM "making a choice", the associated
uncertainty, and the consistency of that choice. (2) We apply this method to
study what moral beliefs are encoded in different LLMs, especially in ambiguous
cases where the right choice is not obvious. We design a large-scale survey
comprising 680 high-ambiguity moral scenarios (e.g., "Should I tell a white
lie?") and 687 low-ambiguity moral scenarios (e.g., "Should I stop for a
pedestrian on the road?"). Each scenario includes a description, two possible
actions, and auxiliary labels indicating violated rules (e.g., "do not kill").
We administer the survey to 28 open- and closed-source LLMs. We find that (a)
in unambiguous scenarios, most models "choose" actions that align with
commonsense. In ambiguous cases, most models express uncertainty. (b) Some
models are uncertain about choosing the commonsense action because their
responses are sensitive to the question-wording. (c) Some models reflect clear
preferences in ambiguous scenarios. Specifically, closed-source models tend to
agree with each other
FED-CD: Federated Causal Discovery from Interventional and Observational Data
Causal discovery, the inference of causal relations from data, is a core task
of fundamental importance in all scientific domains, and several new machine
learning methods for addressing the causal discovery problem have been proposed
recently. However, existing machine learning methods for causal discovery
typically require that the data used for inference is pooled and available in a
centralized location. In many domains of high practical importance, such as in
healthcare, data is only available at local data-generating entities (e.g.
hospitals in the healthcare context), and cannot be shared across entities due
to, among others, privacy and regulatory reasons. In this work, we address the
problem of inferring causal structure - in the form of a directed acyclic graph
(DAG) - from a distributed data set that contains both observational and
interventional data in a privacy-preserving manner by exchanging updates
instead of samples. To this end, we introduce a new federated framework,
FED-CD, that enables the discovery of global causal structures both when the
set of intervened covariates is the same across decentralized entities, and
when the set of intervened covariates are potentially disjoint. We perform a
comprehensive experimental evaluation on synthetic data that demonstrates that
FED-CD enables effective aggregation of decentralized data for causal discovery
without direct sample sharing, even when the contributing distributed data sets
cover disjoint sets of interventions. Effective methods for causal discovery in
distributed data sets could significantly advance scientific discovery and
knowledge sharing in important settings, for instance, healthcare, in which
sharing of data across local sites is difficult or prohibited
Momentum in Reinforcement Learning
International audienceWe adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive q-functions. We derive Momentum Value Iteration (MoVI), a variation of Value iteration that incorporates this momentum idea. Our analysis shows that this allows MoVI to average errors over successive iterations. We show that the proposed approach can be readily extended to deep learning. Specifically,we propose a simple improvement on DQN based on MoVI, and experiment it on Atari games
Trust Your : Gradient-based Intervention Targeting for Causal Discovery
Inferring causal structure from data is a challenging task of fundamental
importance in science. Observational data are often insufficient to identify a
system's causal structure uniquely. While conducting interventions (i.e.,
experiments) can improve the identifiability, such samples are usually
challenging and expensive to obtain. Hence, experimental design approaches for
causal discovery aim to minimize the number of interventions by estimating the
most informative intervention target. In this work, we propose a novel
Gradient-based Intervention Targeting method, abbreviated GIT, that 'trusts'
the gradient estimator of a gradient-based causal discovery framework to
provide signals for the intervention acquisition function. We provide extensive
experiments in simulated and real-world datasets and demonstrate that GIT
performs on par with competitive baselines, surpassing them in the low-data
regime
SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models
The past year has seen rapid acceleration in the development of large
language models (LLMs). For many tasks, there is now a wide range of
open-source and open-access LLMs that are viable alternatives to proprietary
models like ChatGPT. Without proper steering and safeguards, however, LLMs will
readily follow malicious instructions, provide unsafe advice, and generate
toxic content. This is a critical safety risk for businesses and developers. We
introduce SimpleSafetyTests as a new test suite for rapidly and systematically
identifying such critical safety risks. The test suite comprises 100 test
prompts across five harm areas that LLMs, for the vast majority of
applications, should refuse to comply with. We test 11 popular open LLMs and
find critical safety weaknesses in several of them. While some LLMs do not give
a single unsafe response, most models we test respond unsafely on more than 20%
of cases, with over 50% unsafe responses in the extreme. Prepending a
safety-emphasising system prompt substantially reduces the occurrence of unsafe
responses, but does not completely stop them from happening. We recommend that
developers use such system prompts as a first line of defence against critical
safety risks
Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning
International audienceRecent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study